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ABSTHACT 


Ccnsid€rafcl€ research is currently going on into 
the application of distributed computing systems. 
They appear particularly suitable for the computing 
needs of a small marship. The particular constraints 
of the warship's environment are discussed. This is 
followed by a description of how a ring structured 
distributed computing system might be adapted tc 
function in this environment. Included in this 
consideration are the feasibility of attaining 
adequate bus speed, the use of multiply addressed 
messages, and methods of handling real-time 
processing. Of particular interest is the ability tc 
achieve controlled degradation of performance under 
failure, especially failure due to battle damage. 
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I. INTBOD OCTI CS 


With the advance of large scale integration technology 
and the resulting decreasing ccst of digital processing 
eleoents, the concept of using a number of small processors 
tied together by seme form of data bus, rather than using 
one large computer, has been receiving increasing interest. 
The advantages of this type of system include greater 
flexibility in the system, lower costs, and increased 
reliability. Distributed computing systems may gain some or 
all of these advantages depending on the hardware and 
software utilized to bind the processors together into the 
system. 

The advantaces mentioned would be particularly useful 
for application tc a warship's computing requirements. 
Perhaps the foremost advantage in a world of rapidly 
changing technology that makes warships virtually 
obsolescent before tiey can be made operaticnal, is the ease 
with which a properLy designed system could be reconfigured. 
When technigues for dynamically reconfiguring the system in 
the event cf normal failure or battle damage are censideted 
as well, the concept of a distributed computing system 
appear particularly attracti.ve. 

Farber [5,6] has been investigating ring structured 
distributed computing systems for several years. In 
reference 5 he discusses the concept of "fail soft” systems, 
that is, systems which exhibit the property of controlled 
system degradation, rather than catastrophic failure, as a 
result cf component failure. In addition, a prototype ring 
utilizing many of these principles and employing 
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iiicroproc€SS€r tech<nology has been designed at the Naval 
Postgraduate School *(NPS) [ 1 ]. It is not the intent here to 
reiterate arguments regarding the overall advantages of a 
ring structured system with respect to reliability and 
flexibility. These are covered in the references. Hhat is 
intended is to investigate the modifications and extensions 
tc such a system which would be required to adapt it tc the 
computing needs of a small warship. 


A. BASIC CCNCZPTS OP A RING SIEDCiafiED DISIBIBOTED 
CCePUTING SYSTEM 


The heart of the distributed computing system is the 
method of communication between the processors. To gain the 
maximum in flexibility and fault tolerance the system 
should: 

1. avcid centralization of control, 

2. permit communication between processes without regard 
for physical location of the process, 

3. permit execution of processes without regard for 
their physical location. 

The last two ease the job of dynamic reconfiguration of 
the system while the first is obviously necessary to prevent 
the failure cf a particular component from bringing the 
whcle system to a halt (catastrophic failure). To achieve 
these twc goals the following basic system is proposed. 

The ccmmunicatipn network consists of a unidirectional 
ring to which a processor is attached via a ring interface. 
Only the ring interface in control can transmit a message. 
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On completicn, control is passed to the next ring interface. 
Thus, ccntrcl is decentralized. Independent timers in the 
interfaces ensure that no one ring interface monopolizes the 
ring and that if control is lost, it will be regained ty one 
of the other ring interfaces. 

The indication that a ring interface may take control is 
the receipt cf a special message called a control token. 
Hhen an interface has no message to transmit, it simply 
passes the token on to the next ring interface. If it has a 
message to send, the control token is not retransmitted. 
Instead, the message is placed onto the ring, is passed 
^completely around the ring and removed by the originating 
xing interface. The control token is put back onto the ring 
after the end of the message. 

Two tokens, the start of message (SCM) and end of 
message (fCH), are used to delineate the data being sent. 
The SCH is followed by the name of the addressee. The data 
comes next and then the EOM token followed by several tits 
used to check for the proper receipt of the message. 

Each ring interface has a list cf the processes which 
are operating in its host processor. Messages are addressed 
to these processes. If the ring interface finds a match 
between the address on an incoming message and one cf the 
processes in its list, it passes the message to its host 
processor as well as passing it on around the ring. Status 
bits appended to the end of the message as it is passed on 
indicate to the originating ring interface whether or not a 
message has been matched and accepted during its 
curcumnavigation of the ring. This accomplishes the second 
objective, communicating between processes regardless of 
their physical location. 

finally, resource allocation is accomplished by a system 
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of reguests for bids. If a new process needs to be set up, 
a reguest for bids describing the resources reguired is sent 
around the ting, ladh processor having those resources will 
reply with a bid giving its present resource availability 
state.. The reguestox of the bids will then select the most 
appropriate bid and confirm the new allocation with the 
accepted bidder,. Hence there is a method of dynamic 
reallocation of resources. 

This, then, is the basic distributed computing system 
which will be considered for adaptation to the small 
warship’s needs. 1 more detailed description of this design 
can be found in references 1 and 2. 


B. HABCHAEE CCNSICEHATIONS 


As can be seen frcm the previous discussion, the heart of 
the ring structured distributed computing system is the ring 
interface. It must have the capability to recognize the 
various tokens, maintain lists of processes, and convert the 
sequential data format of the ring to the format reguired by 
its host. The basic difference between the rings designed 
at NFS [1] and at the University of California at Irvine [6] 
is the technology employed in construction of the ring 
interface. 

The ring interface at Irvine is hard wired. As such it 
enjoys the advantage of greater efficiency of design and 
hence higher data transmission rates. However, it loses in 
fleiibility and standardization. It is designed to interface 
with one particular host and to adapt such a ring interface 
tc a different host is virtually impossible. Moreover, the 
resulting differences in ring interfaces designed for 
different hosts can pose a maintenance problem. 


10 



.rtJ 

n 



- Wf 
• 3rii 
a:«: 


•» 

'•4? 

c»e 

5 / 

?» 

ti 

Ki 


'‘W 

>1 

II 

« 

I 


» 


The ring interface at NFS is designed around a 
microprocesscr,. The ring speed in this case is limited by 
the firmmare of the programme stored in its programmable 
read only memory (FROM)- and the access time of this memory. 
For the greatest flexibility the NFS ring employs an 
erasable PECfi. Thus if the ring protocol were to be changed 
or a new host required a different format for exchange of 
data, the microprocessor memory could br erased and a new 
set of firmware written. This entails a sacrifice in speed, 
however, as erasable FBOIl*s have slower cycle times than 
non-erasatle cnes. By using a non-erasable FROM the speed 
can be increased but a change would entail replacing the 
memory with a different chip containing the new programme. 
In both these cases the microprocessor has the advantage of 
standard hardware fpr the ring with only a variation in the 
firmware to accomodate the various hosts. 

Flexibility is of particular value in the shipboard 
application where a large number of the hosts may be "dumb”, 
such as serve system contrcllers or monitoring devices, or 
may be in existance at the time of system design. Because 
of the advantages in maintenance and flexibility, the 
microprocesscr technplogy will be considered here. The 
problems of ring data transmission rates will be addressed 
again after the requirements of the shipboard application 
have been discussed. 
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II- ANALYSIS OF SYSTEM fiSQUIRE MENTS 


A, EXTESI OF SYSTEM 


Th€ first question which must be addressed when 
considering system requirements is the extent of the system. 
Hhat functions of a warship should be included for 
processing in the distributed computing system? There are 
some areas that are fairly obvious, i.e. those that are 
included in current command and control systems. These 
include evaluation and processing of senscr information 
(radar tracking, electronic warfare evaluation); tactical 
data processing (maintenance of track files, files of 
standard tactical prpcedures, etc.); tactical communications 
(LIMK); fire control calculations; tactical displays; 
navigation routines (ship’s position, closest point of 
approach calculations, course to intercept, etc.). But 
there is no reason to limit it to these. 

Digital computer control of main engines and auto-pilots 
are already in existence. Including these in the 
distributed computing system would facilitate automatic 
control cf emergency and evasive manoeuvres e.g. 
man-cvertcard, torpedo evasion; and also the automatic 
implementation of tactical manoeuvres e.g. lost contact 
searches, zig-zag courses, station keeping. Eemote 
monitoring and automatic alarm systems for auxiliary 
machinery exists as a subsystem of the main machinery 
control. Expansion of this to monitoring other ship's 
conditions, smcke alarms, flooding, etc. , and making it 
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part of the distributed computing system would provide an 
effective and flexible damage control monitoring system. 

Finally, communications is a fruitful area for 
automation. Tactical data is already encoded, transmitted, 
decoded, sorted, displayed, and stored automatically. The 
problems of extension to the remaining communications nets 
do not sees overwiielming and the possible savings in 
personnel, paper and time appear to be worth-while. In 
addition, the use of a computer for frequency assignment 
algorithms would be n distinct advantage and if this were 
coupled with direct control of transmitters and receivers, 
electromagnetic emission control would be greatly 
facilitated. Making this computer part of the , ship's 
distributed computing system would allow such problems as 
the efficient handling of priority traffic to be easily 
attacked. 

Such an extended system is under development by the 
Canadian Armed Fordes under the title of "Shipboard 
Integrated Erccessi<ag and Display System" [7,9]. It is the 
adaptation of a ring structured distributed computing system 
to the requirements of this system that will be considered. 
Most of the leguirements in the following sections are based 
on the preliminary requirements for the Canadian system [9] 
and are in agreement with these requirements. 


B. DETAILEE SOBSISTEM BEQOIfiEMFNTS 


1. Eisplays 

The display reguireiments for various departments on a ship 
vary widely. Communications can be satisfied with purely an 
alpha-numeric display. To the engineering department, a 
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graphics capability would be a desirable feature. The 
greatest demand undoubtedly comes from the tactical displays 
which are recuired tp present raw video from various sensors 
as well as the synthetic graphical and alpha-numeric data 
that are produced ijy the various processors, i.e. smoothed 
tracks, track designators, messages, etc. 

ill these requirements will affect the decisions 
with respect tc display technology, i.e., what are the 
advantages and disadvantages of raster scan versus vector 
generation versus dot matrix presentation, etc. However, 
with respect to the system design of a processing and 
display system , most of this is irrelevant. What is 
important is the "intellegence" of the display, for it is 
this characteristic that will determine the type and 
quantity of data that must be passed to a display device to 
maintain the presentation desired by the operator. 

An allied aspect of display technology is the method 
cf maintaining data on the display - whether a long 
persistence cathode ray tube should be used or the data 
should be freguently refreshed. The latter is the only 
practical sclution for any sort of dynamic display and for 
the rest cf the discussion it will be assumed that the 
display must be refreshed at least 30 times a second. 
Refresh rate then provides a critical frequency in the 
inplementaticn cf the display system. 

An important differentation can now be made in the 
types of data which are to be displayed, i.e. those data 
which may be expected to remain constant for more than one 
thirtieth of a second and those that may not. For the latter 
it makes no difference whether the display is "intelligent" 
enough tc refresh itself or not. The data will have changed 
in the interval between successive refresh cycles. 
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Jcr the Ifiss volatile data it is a distinct 
advantage for the display to be able to store the data for 
the present picture. It is then possible to send to the 
display only changes to the existing picture. Otherwise, 
the data for a ccmplete picture must be transmitted to the 
display everj refresh cycle. 

Imposing this distinction on the types of data the 
ship's system must display, raw sensor data fall intc the 
volatile category while the synthetic data are more stable. 
While some displajs are required to present raw sensor data, 
all have a requirement for presenting synthetic data. It 
will be assumed then that self-refresh is a desirable 
characteristic and will be incorporated into displays used 
in the distributed computing system. 

While the desirability of a self-refresh capability 
is fairly easj to shpw, the case for further increases in 
display intelligence is not so clear cut. Should the display 
have the ability to jnanipulate data? At one extreme one can 
conceive of a tactical display that wculd contain all 
synthetic data out to the maximum range it can accomcdate, 
and all requests for lesser ranges, filtered data, etc. 
could be handled within the display itself. This would 
allow all update messages to be broadcast to all displays 
rather than having to be tailored to each display's current 
presentation and, hence, individually addressed. Also, such 
a displaj wculd require very little in the way of 
communication from the display to the system since there 
would be no need fpr the system to be aware of each 
display's particular presentation. It would, however, 
require a large mempry and considerable computing power 
within each display. At the other extreme might be a simple 
routine sc that several sequential iterations of data (sonar 
range sweeps, frequency scans) could be displayed such that 
as each iteration was displayed the remainder move up and 
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tb€ oldest is discarded. Intermediate between these two 
eitiemes might be the ability to identify groups of data on 
the screen aid move the groups by incrementing the position. 

It is apparent that actual data rates are very 
dependent on the type of data to be presented and the amount 
of processing that can be done within the display. The NATO 
Industrial Advisory Group £ 8 ] is using an average of 16,000 
bits per second as the communication requirement for a 
self-refreshing display. 

2• Acti ve Se nso rs 

The active sensors employed in a modern warship are 
radars and active soaar. Even in a small warship there 
could be a minimuin of four to six radars; search, fire 
control, and navigation; as well as one or two active 
sonars, hull mounted and variable depth. The raw data rate 
of a radar set can be determined theoretically by the 
sampling rate required to capture all the information 
available in the bandwidth of the intermediate frequency 
amplifier chain. Since the IF bandwidth is in turn a 
critical function in the optimization of the design of the 
radar, the data rate can ultimately be related back to the 
operational parameters of the radar. For a typical search 
radar, pulse width pne microsecond, the IF bandwidth is one 
HHz and the data rate required to ensure no less of data is 
thus 1.5 to 2 million bits per second. A high definition 
radar with its shorter pulse width would obviously require a 
higher rate. Sonar, due to the more liesurely velocity of 
propogaticn of its energy, has a much lower raw data rate of 
85,000 to 20C,000 bits per second. 

It is apparent that to multiplex one, let alone 
several, active sensors on a general usage data bus would 
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impose a rathei severe load unless the bus speed Mere very 
high. However, the requirement to display taw sensor data 
for evaluation by an operator must be met. 

Neat to be considered is the effect of processing 
the raw data. The f|rst level of processing would appear to 
be the autcaatic detection of contacts. This would involve 
passing only data cn signals that exceed a certain level. 
Here again a typical situation will be postulated to obtain 
ar idea cf required data rates. Assume a radar rotating at 
60 rpm and tte possibility of 200 valid contacts. To obtain 
a reasonable probability of detection, the threshold level 
for signals tc be aiccepted must be set fairly low resulting 
in a large number of false alarms which must be filtered out 
by sweep tc sweep correlation. Thus, the number of contacts 
accepted per scan would be in the order of 500 to 1000, On 
each of these a minimum of range and bearing, and sweep or 
time data must be passed. Even if only 10 tits of precision 
are used, the resulting bit rate reaches 15,000 to 50,000 
bits per second. 

Einally, the data rate resulting from autc-track 
processing can be evaluated. Here, only valid contact data 
are ttansoitted and these would include target 
identification, x and y coordinates, time, course and speed. 
Additional data need only be sent when there is a change. 
Total data per track may be in the order of 200 bits but an 
update message would be considerably less than that, and 
wculd be required much less often than once per track per 
scan. Although the instantaneous communication requirements 
wculd be much more variable, the average rate would be much 
lower, something in the order of 4000 bits per second for 
the assumed criterion of 200 valid tracks. 

The discussion of processed data sc far has been 
limited tc radar data but it should be apparent that this is 
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tb€ limiting case. Sonar contacts would te handled in the 
same manner let the expected number o£ underwater contacts 
is in the order p£ one or two, and ten cr more would be 
unlikely. The communication load would then te in the order 
of tens cf hits per second. 

Here is one further area of processing of active 
sensor information that needs to be considered, that of 
multi-sensor correXation. While autodetection might be 
carried cut with a microcontroller, auto-track processing 
has a sufficiently great computing requirement that a 
minicomputer would, reasonably be employed. Why not then 
extend the process to compare the contacts of various 
sensors to determine if the same object had teen detected by 
more than ore senspr? With appropriate processing and 
feedback tc control the sensitivity of the sensors, greater 
detection probabilities can be obtained in addition tc the 
immediate result of providing more accurate information by 
combining the data from independent sources. This would 
have the additional benefit of further reducing the traffic 
on the communication net since redundant information on the 
same contact tut originating from more than one sensor would 
be eliminated. However, the magnitude of the reduction 
would te quite small, particularly in relation tc those 
obtained in going from autodetection tc autc-tracking. 

3• f^ssive Se n sors 


Passive sensors include acoustic detectors as well 
as electromagnetic energy detectors for frequencies from 
that of radar to that of visible light. The latter can be 
broken into two grpups on the basis cf the characteristics 
of their output data^ Electronic warfare sensors cover the 
frequency hands from roughly one to fifty GHz. There is a 
considerable amount pf preprocessing done and the raw data 
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are of little or no interest. The second group, optical 
s^steios, produces a linescan video image as raw output. 
Included in this group are low light level television and 
infrared sensors. As the data from these sensors is 
compatible, processing may include multi-sensor correlation 
as well as digital enhancement of the image. The computing 
reguirements for this are rather high and would probably 
reguire a dedicated minicomputer. 

Now to consider each of these three sensor groups 
individually. Passive acoustic data is normally presented 
for human evaluatipn after preprocessing to obtain sound 
intensity versus frequency versus time information. The 
most common presentation method is as an intensity modulated 
frequency sweep with the past data being shifted along the 
time axis as each new frequency sweep is started. An 
alternative coming into greater use is to construct a three 
dimensional graph .with the time axis appearing to recede 
into the screen. The latter would change less often, a new 
sweep being added every 2 to 5 seconds, whereas the 
intensity modulated display would initiate a new sweep 4 to 
8 times a second. 

If a systematic shifting routine were available, as 
discussed in section II.A.1, the update would require in the 
order of 5C00 bits of information for the intensity 
modulation and 3^0,000 for the graphical plot. The 
resulting data rates would then be 20,000 to 40,000 bits per 
second fcr the intensity modulated method (5000 bits 4 to 8 
times a second), or 60,000 to 150,000 bits per second for 
the graphical plot (300,000 bits every 2 to 5 seconds). 
Having to update the whole screen each time would increase 
the data rate to several million bits per second. 

The optical systems present a volatile image that 
will change at a rate higher than the 30 Hz refresh rate. 
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Here the criterion of IF bandwidth can be applied as in the 
radar sjstei and data rates of several nillion bits per 
second result. 

In the cas.e of both passive sonar and the optical 
systems much further processing can be done, including 
correlation with contacts held on other sensors both active 
and passive. Ihe communications load produced as a result 
of this is of the nature of the update and amplifying 
material cn valid contacts, as discussed in section II.A.2. 
Thus a data rate in the order of 10 to 100 bits per second 
uculd be reasonable. 

E.H. sensors, as mentioned, produce a highly 
preprocessed output nith the raw data being cf little use. 
The computing regu^rements are quite high, undoubtedly 
requiring a dedicated processor. However, the communication 
requirement would again be that of passing update material 
and 500 bits per secpnd would be adequate. 


h. *€!£0ns 


Ihe weapons putfit of a small warship would consist 
cf a mii cf missile launchers and guns and "soft weapons'* 
such as jammers and chaff launchers. For remote control of 
servo devices data is usually supplied 32 times per second. 
Even for the most complicated launcher a few hundred bits of 
data should provide adequate information at each update and 
100 bits would be a reasonable average. This results in a 
3200 bits per second average rate for each weapon and even 
with 8 to 10 weapons operating simultaneously the total 
average rate would be 32,000 bits per second. 

It must be considered, however, that these are real 
time systems and the timing of the messages is critical. It 
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will ther€£ci6 be necessary to devise a method of ensuring 
that weafons control messages do not get delayed i.e. left 
waiting in octput gueues. 


5. cc mmcn icatioo s 


Communications has one area in which the needs are 
already well defined. Computer controlled tactical data 
ccmmunicatici: via LItliK 11 has been in use for several years 
and 15,000 bits per second could be considered a reasonable 
estimate of tcs loading by this system [4]. 

Ihe reguireaents for automated handling of general 
message traffic are more obscure. Data exist on the number 
of messages per day and can be broken down by priority and 
classification. Storage requirements can thus be easily 
determined, four million bits per day should suffice and two 
days’ messaces should be immediately retrievable £9]* 
Messages more than two days old can be relegated to cff-line 
storage. 


Ihe effect pn data bus loading is considerably less 
clear. It can be reasonably assumed, however, that with the 
exception of priority messages, general traffic will be read 
in periods of relative leisure, hence, in periods of low 
system activity. Ihe effect of this on bus loading should 
then be minimal. friority messages would go cut on the bus 
immediately but this type of traffic is measured in messages 
per hour if net per day and in terms of bits per second 
would seem tc be insignificant. 

There is one other consideration and that is the 
case where the direct-access mass storage for current 
messages is a separate node on the bus. In this case all 
message traffic would put a load on the bus as it was 
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transferred tc the mass storage device. Most traffic is 
teletype at 120 characters per minute cr 16 bits per second. 
Sven with siiultaneous sending and receiving on several 
frequencies, the bus loading would be in the order cf 100 to 
200 bits per second. 

fto£)Jlsion Control 

Az mentioned previously, the main engines of some 
ships are already controlled by digital computers, and many 
ships are steered by autopilots. The step tc digititing the 
engine and steering orders and routing them via the ship 
data bus is a small pne but opens up the pcssibility of many 
desirable features. Instead of having someone drive the ship 
arcund a lest contact search pattern displayed on his 
computer terminal, the computer can do it a.utomatically. If 
a torpedo is detected cn sonar, evasive action could be 
initiated automatically or by human intervention with 
automatic execution. 

The cost pf this would be rather small. The 
frequency and length cf propulsion commands is very low, 20 
tc 50 hits not more often than every 2 to 5 seconds. 
Therefore, peak loading would conceivably be 50 bits per 
second. 


The machinery control computer would itself need to 
mccitor status infprmation from the machinery. This would 
involve in the order of 100 points per second with 8 to 12 
bits of data apiece [7], These 1200 bits per second would 
not produce much of a load on the bus, however, for other 
reasons it may not be feasible to use the ship's bus for 
this information. TJiere is obviously a need for a great 
deal of autonomy in the propulsion control system. As with 
no other, it is critical to the survival of the ship. The 
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ad\>antage£ in makijjg it part of the ship's distributed 
ccaputing system are features that are nice to have but not 
critical. Ihere is, therefore, a strong argument for 
dedicating tte propulsion system completely and only using 
the bus for these npn-critical features, particularly since 
it would still permit the excess capacity in the machinery 
control computers to be used by other systems. 


7• Ship Mo nitoring 


Dnder this heading will be included all the various 
miscellaneous devices throughout the ship that provide 
information on the state of the ship. Included are existing 
devices such as gyrp-compasses and stable elements, and 
ether things which may not be remotely monitored at present, 
such as the state of auxiliary machinery, hull and fire 
pumps and air conditioning units, and emergency warning 
devices, flooding indicators and smoke alarms. There could 
conceivably he from several hundred to several thousand such 
devices. Some small number, less than 20 and including log 
and compass, would require several bits of information and 
monitoring several times a second. The vast majority would 
require 1 tit and monitoring every few seconds to few 
minutes. Kith agpi^opi^iate preprocessing and data 
compression ty microcontrollers before being put on the bus, 
the additional loadiaig would be in the order of 1,0C0 to 
2,000 bits per second to monitor 1,000 points. 


C. SOMMABI 


In the pieceding sections of this chapter the 
requirements of the various individual subsystems 
discussed. These data are summarized in table 1. 
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TABLE 1 


£DBSYSI£M COHMONICATION HEQOIBEHENTS 

Subsystem Bits/Seccnd 

Displays. 16,00C 

Active Sensors 

Bacar rav.^. 2,00O,00C 

prccfissed..... 4,00C 

Scnar rau.^. 150,000 

processed.... 2C 

Passive Sensors 

Scnar rav.,. 75,000 

prccessed. . ... 50 

Optical ra.u... 4,000,000 

processed..... 50 

E.S. 500 

Heapcns 

Ccctrol of a single weapon......... 3,200* 

Conaunications 

Tactical...... 15,000 

General. <10 

Propulsicn Control........... 50 

Ship hcnitoring. .. .... 1,000 

*Beal-time requirement 
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emphasis has been on interprocessor communication or bus 
loading, since this is the critical requirement in a 
distributed ccmputijag system. The figures in table 1 are 
valid as long as, at a minimum, the computational 
requirements of any individual subsystem can be met in a 
single processor. Several processes active in one processor 
might decrease, tut would in no way increase, the bus 
loading. 

The question of how much computing power is required for 
the ship’s sjstem has had little attention. It is obviously 
an important problem but it is one which would require a 
much more detailed analysis of the various function than has 
been attempted here. It is not intended to discuss this in 
depth, but a few general comments can be made. 

The processing requirements can be met by choosing the 
the appropriate number and size of computers. While a 
distributed computing system puts no constraint on the 
maximum size of computers to be used in it, one of its main 
advantages is to he able to use several smaller and, 
therefore, cheaper computers, rather than a large expensive 
one. The oinimum si 2 e, as has been mentioned, is that which 
can handle the processing requirements for a single 
subsystem. Having ejcceeded this minimum size, the number of 
computers on the bus is transparent to the user and the 
software. Therefore, the system designer is basically free 
to vary the number and size of processors as he wishes. 


D. DEIAIIHC £ISTi£ BEQOIHEMENIS 

Having considered the individual requirements of the 
various subsjstems and components of a distributed computing 
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TABLE 2 


DATA DISPLAY fiEQOIREllBNlS 


fUQCtiCO 


Comaand aad ccntrol 

E.H. 

Sonar, active 
passive 
iieapcQS 

ComaunicaticDS 
Propulsion ccntrol 
Ship aonitoricg 


CRT Displays 
number raw synthetic 


10 

1 

2 

1 

3 

2 

2 

2 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 

X 


Hard Copy 


1 


TABLE 3 


SYSTEM REQUIREMENTS 
Subsystem Bits/Second 


Displays 23 X 16000 368,000 

Processed xadar 4,000 

Processed sonar 20 

Passive sonar 50 

Optical 50 

E.i. 500 

Heapcns 6 X 3200 19,200* 

Ccamunications, tactical 15,000 

general 10 

Prcpulsicn control 50 

Ship mcnitpring 1,500 

TOTAL 408,570 


♦Real time reguirement 
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system, it is necessary to consider the system as a whole to 
determine the magnitude of the system (number of nodes, 
aggregate bus loadj-ng, etc.) and any system constraints. 
Table 2 summarizes a typical data display reguirement fcr a 
small general purpose warship. Since raw sensor data rates 
are such that a separate dedicated bus will be required, 
those displays which must handle this data are identified. 
Table 3 is a list of all systems tied to the general purpose 
bus with their estimated average communications 
requirements. The individual requirements are summed tc 
give a tctal average bus lead of 400,000 bits per second. 

Note, however, that this does not include communication 
requirements associated with operating system overhead. This 
reguirement would have to be determined as the system design 
evolved. Also these are average figures, peak loading could 
be much bigh€r and tie effect of real-time control messages 
must be considered. 

There is a requirement to look at the various subsystems 
from the point cf view of autonomy or dedication. Do some 
subsystems because of their importance or unique 
requirements warrant being configured as autonomous system:?? 
Are there groupings of subsystems that might have an 
autonomous function? The importance of main machinery 
control has been mentioned and is considered sufficiently 
great tc justify autonomy, probably requiring 100S 
redundancy cf processors and complete cross coupling. This 
is the only case tiat is sufficiently unique and impertant 
for such treatment. 

The arguments $or grouping of subsystems are somewhat 
less clear. There are several such groups possible, 
antisubmarine weappns and sonars, antisurface/anti-air 
weapons and radars, E.W. and jammers and decoys. A case 
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could b€ presented for making these groups autcncmcus, 
however, there is also considerable interchange of 
information amogg these systems. The technical aspects of 
system integration and reliability have a bearing on this 
decision and it uill be considered in more detail later. 

Finally, system reliability must be considered. 
Controlled degradation of performance under failure is one 
of the greatest potential advantages of the ring structured 
distributed computing system. However, previous ucrk by 
Farber and Harris has not had to contend with one of the 
most obvious failure modes in the military environment, that 
of battle carnage. The ability of the system to continue to 
operate under these conditions is of utmost importance. 
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III. SYSTEM DESIGN 


A. ABEAS BECOIfilNG CONSIDEBATION 

The lecuirements of a warship's computiag systen impose 
several restraints <not allowed for in the systems developed 
ty Farber and Harris. These are: 

1. bus speed, 

2. real time processing, 

3. multiple addressees, and 

4. battle damage. 

The effects of each of these on systems design will new be 
individually discussed. 


1. Eus f reed 


In a warship!s computing system of the nature that 
has been discussed, it is critical that the bus be able to 
support the data rate required with little or no delays. In 
a system such as designed by Farber, if the bus should 
beceme overloaded for a period of time the result is mainly 
inconvenience to the user. On.a warship such an overload 
could be fatal. The bus data rate is, therefore, an 
important factor in system design. 
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Tlie t\i£ data rate is determined by the node design 
and the propagation characteristics of the bus itself. It 
is these areas that mill now be discussed. & node consists 
of a repeater and a ring interface. Ihe bus consists cf the 
repeaters and their g.nterconnecting wiring. 

Khile the possible media for connecting the nodes 
together are numerous, from a single wire with ground return 
to optical fibres, only three are worth considering due to 
problems cf electromagnetic interference in the shipboard 
environment. Ihese are coaxial cable, shielded twisted wire 
pair, and optical fibres. All three are capable of 
transmitting at five to ten megabits per second over the 
cable lengths anticipated on board ship. Optical fibres are 
capable of much higher frequencies. 

Ac independent problem that arises when connecting 
several computers together electrically is establishing a 
common electrical ground. lo avoid this common ground 
problem when using shielded electrical cables, optical 
isolators can be inserted into the bus at each node. The 
use of optical fibres, of course, dispenses with this 
problem altogether- Since cost, availability, and 
convenience cf optical fibres are rapidly approaching those 
of coaxial cable, they would appear to be the best choice 
for the bus. 

Hbile the ring operating at the University of 
California at Irvine operates at 2.25 GHz, it uses, as 
mentioned, a hardwired interface. Ihe ring interface 
proposed by Harris [1] using a microprocessor was designed 
to operate at 112 kilobits per second bus data rate. Ibis 
was determined by the memory cycle time (1.1 microsecond) of 
the erasable EfiCM in the ring interface microprocessor, a 
requirement for the execution of four microprocessor 
instructions between bits, and a two for one encoding ratio 
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of transnitted bits to data bits. The remainder c£ the 
circuitry, being transistor transistor logic (TTL) modules, 
is capable cf sustaining five to ten megabits per second 
with no difficulty. Currently non-erasable PBCH's are 
available with access times of the order of 50 nanoseconds. 
This would allow the bus data rate to be increased tc 2.5 
meoabits per second. 

The leguiremsnts in table 3, for the system being 
considered, result in an average bus data rate of 4C0,000 
bits per second. The most significant load (more than 90% 
of the total) is that required for displays. Even allowing 
for an error in the estimate of the average data rate of a 
factor cf three, a 2,5 MHz data bus still has sufficient 
capacity to handle peak loads of two times the average. If 
this is not capable of handling the communications 
requirements, PROMs utilizing emitter coupled logic are 
available with 20 ns cycle times raising the possible bus 
rate to ever six MHz, 

The ether side of the interface must new be 
censidered, the interface between the nede and the 
processor. Fcr purposes of discussion the characteristics 
of the AlI/OYK-20 minicomputer will be used, as it is a 
militarized minicomputer in cemmon use. This computer uses 
a 16 bit I/O word. Therefore, at 2.5 million bits per second 
bus rate with 16 bit buffering in the node the I/O 
ccntroller would require access to memory at least once 
every 6.4 iiicreseepnds or about every eighth memory cycle. 
The computer can thus handle this date rate continuously 
with about a 12.5% Ipss of processing time. 

If the bus rate were raised to five cr six MHz the 
loss of processing time would increase to 25 or 30 per cent 
for continuots sending and receiving. This still may be 
acceptable if, in fact, a lower proportion of the time is 
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spent in ccmmunicating. With 10 to 15 processors anj one 
should be ccoiBunicating with the bus only 15 to 25 per cent 
of the tioe cn the average. Thus, direct communications 
with the ring interface from a five MHz bus with only cne 16 
bit serial tc parallel conversion would still be adequate. 

If bus ccmunication requires tco much processor 
tine, there are several possible courses of action. First 
the AIi/OIK-20 is capable of 32 bit parallel communication. 
This would halve the time required to service bus interrupts 
but the buffer size in the ring interface would have to be 
increased. Second, a direct memory access capability is 
available in the AN/UIK-20 which would allow the ring 
interface to access memory by a second port without locking 
out the central processor. However, this would require 
oemcry interface log^c in the ring interface. Finally a 
first in first out (FIFO) buffer in the ring interface would 
allcw more flexibility in the rate of passing data between 
the ring interface and the host processor, i.e. the 
processor could, be interrupted once and a block of several 
words could be accepted or transmitted on consecutive memory 
cycles. Ihe advantage gained by this technique would be 
strongly dependent on the nature of the processing being 
carried cut as well as the relationship between maximum 
message length on the bus and the size of the FIFO buffer. 

Sc far no mention has been made of disseminating raw 
data. Ibe requirement to display this data is present in 
approximate!j 16- displays (see table 1). It should be 
apparent from the discussion of the requirements that a 
dedicated bus is required for this information. Since a 
single display would only be required tc display the raw 
output frcm cne sensor at a time, cne obvious configuration 
for this bus would be a ''star" pattern. All sensors would 
feed intc a central switching unit which would then 
distribute the data to the various displays as required. As 
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this raw data system is independent of the distributed 
ccmputing system and would be required no matter what type 
of computing system were used, there will be no further 
discussion of this requirement. 


2* Hea l Time Processing 


The transnission of messages for real-time control 
purposes poses another problem. If such a message were held 
overly Icng in a queue waiting for the ring interface to 
gain ccntrcl havcc could result. Therefore, there must be a 
method whereby a real-time message can be forced onto the 
bos. 


Cne possible method could be to have a priority 
interrupt tchen. A process requiring to transmit a real-time 
message cculd interrupt the ring, transmit an interrupt 
token to warn the rest of the ring and the sender of the 
interrupted message that it had pre-empted the ring, and 
then transnit the priority message in the normal manner. 
The simplest foLlow up would be for the interface 
originating the interrupted message to retransmit it at the 
next opportunity, the same as if it had detected a faulty 
transmission on an uninterrupted message. A possible 
problem arises from this system, however, in that control 
has jumped over the nodes between the interrupted sender and 
the interrupting node. Thus, instead of being guaranteed a 
chance to seed a message at least once in a time period 
equal tc the product ot the number of nodes and the longest 
message time, a node could be locked out. Tc preclude this 
the interrupting interface, rather than placing a control 
token on the ring at the end of its message, could place a 
special marker ontp the ring which when recieved by the 
interrupted node would signify that it could reassert 
control and retransmit its message. 
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inotl!6£, but more complicated way uould hav€ the 
interrupting ring interface store the remainder c£ the 
interrupted message and retransmit it cn completion of the 
priority message. This would have the least detrimental 
effect on ring communications but would reguire either a 
FIIO buffer in the ring interface or full duplex operation 
between the ring interface and its host processor. A 
variation cn this would be to impose a fixed and fairly 
short length cn priority messages. A serial-in serial-out 
shift register would then be used to delay the interrupted 
message the appropriate number of bits. This is perhaps the 
most pratical of the methods discussed. For example a 48 
hit delay would allom three twelve-bit control words plus 12 
bits for interrupt token and addressing. The end of the 
priority message could then be found by counting bits rather 
than needing another token. 


Bu ltir le Ad dressees 

Cne of the basic tenets of the ring structured 
distributed computing system is that interprocessor 
communications are addressed to processes, not processors. 
In the ship's general purpose distributed computing system 
there are a number pf cases where the same data is reguired 
by several processes^ for example: ship's course, roll and 
pitch are reguired for processing of sensor data, for gun 
control calculations, for manoeuvring etc.; also, an update 
for synthetic video may be used by several displays. To 
mitimize bus loading problems, it is desirable that one 
message be sufficient to transfer such data to all users of 
it. There are several possible ways of accomplishing this. 
One would be to set up a specific receiving process for each 
of these various classes of data. However, this would mean 
that a process which requires this information must ensure 
that the appropriate receiving process were resident in the 
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sane ptccessor. This would violate the teguireiaent for 
Independance of picc£sses and physical locations. 

A second ppssibility would be tc structure the 
process Dames to allpw qualifiers. A general message could 
then be sent to all processes with the same qualifier. 
Problems arise here if the number of qualifiers is large, 
then complicated decpders and/or long process names would be 
needed. 


Perhaps the simplest solution would be to allow a 
process tc have several names. Some of these would be the 
same as names of other processes requiring the same 
informaticn. Host spftware would only have to ensure that 
all resident users of the same name were referred to the 
same input buffer for the common data. 


fau lt To lerance Under Battle D amag e 

A situation that Barber's distributed computing 
system cannot cope with is that of a failure in the ring 
itself. Shile the failure of a twisted wire pair is so 
improbable as to be inconsequential in a non-combat 
environment, the probability of the shipboard ring being 
severed by tattle damage is very real and cannot be ignored. 
The scluticn requires three separate actions; 

1. determination that the ring has teen severed, 

2. localization of the break, 

3. repair or bypassing of the break. 


The severing of the ring will be immediately 
apparent tc the npde immediately beyond the break. Using 
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the same encoding scheme as Harris [1] ensures that the 
input cannct remain in the same state for more than two 
deck pulses. The only exception is for a token, at which 
time the state persists for three clock pulses. Detection 
of four cr more identical bits in a row would indicate a 
break. If the detecting node then immediately started to 
transmit, no ether npde would detect the error and the first 
twe steps cf the solution would have been accomplished. 
Rhat happens next will depend on the hardware configuration. 

It is obvious that some sort of path redundancy must 
be built into the ri<ng to allow for this situation. The 
simplest would be to have two or three cr more complete 
rings. fer reasons to be discussed later they should 
connect the nodes in different sequences. All nodes would 
listen tc all incoming connections but only transmit cn one 
output connection, all others being held in the same state. 
If a break were detected the detecting node could send on 
the broken ring a priority message which would cause each 
node to switch to the next back up ring. Simultaneous 
breaks wculd be handled by this system since each node 
immediatly dewnstrean of a break would initiate such a 
message and it would be passed along tc all nodes until the 
next break was reached. The time'-out system would then 
reinitialize the rinq. If the new ring were not established 
within a specified Ipnger time limit an automatic switch to 
the next ring could be incorporated. 
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Figure 1 - A SIMPLE RING 
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Ibe sain wfakness in this system is the situation 
where a node is disabled. This would interrupt all rings 
since they all pass through all nodes. Furtheriaor6, if all 
rings connected the nodes in the same sequence, the loss of 
a node is catastrpphic. If the nodes are connected in 
different sequences, ways can be devised to use segments of 
various rings to bypass a defective node. The following 
paragraphs propose one method of solving this problem. 

Figure 1 shows a seven node ring with each node 
having three input and three output connections. The circle 
connects the nod.es in one sequence (ring 1) . The 
connections shewn outside the circle (ring 2) skip every 
ether nede cn the circle, and the connections shown inside 
(ring 3) skip two nodes. For this arrangement, as long as 
the number cf nodes is not a multiple of two or three, rings 
1, 2, and 3 will be independent and complete. Dummy nedes 
can be inserted into the ring to ensure that this condition 
is met. 


New consider what happens if ring 1 is in use and a 
break occurs between nodes three and four. All interfaces 
are listening to all inputs but transmitting on ring 1. All 
inputs tc node 4 are now constant and it will realize that 
ring 1 has been broken. One of two things will occur at 
node 4 at this point.* if node 4 has as a host a general 
purpose processor, it will alert the processor and send out 
on the ring a ring broken (HER) token and a message saying 
to stand by for instructions. If it has a less intelligent 
host (a gun controller or a mass storage device perhaps), it 
will transmit an REH token and its node number. In this 
case the first node having an appropriate hest will alert 
its host and change the RBH message to ‘'wait for 
instructions”. Once an RBR token has been received a node 
would disregard a constant input until after the ring had 
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been reestatlished. This is necessary to avoid having 
messages arriving at a node on two different inputs at one 
time. 


The situatipn is now that the first general forpose 
processor downstream of the break knows where the break is. 
Moreover, since the prder of nodes on the working ring would 
be stored by all processors each time the ring is 
established cr res.tructured, the processor would be in a 
position to control the restructuring of the ring. 

Obviously, the first thing to try is a switch to an 
alternate rirg. In the example, node 5 would take control 
and attempt to switch to ring 2. It would transmit this 
command on ring 2 and each node, as it received the command, 
would retransmit it on the new ring. Khen the message 
returned to node 6 it would know reconfiguration was 
ccaplete and reinitialize the ring. 

If ring 2 were also broken, the message sent cut by 
node 5 would never return to it. after an appropriate time 
node 5 wculd transmit a command on ring 3 to switch to ring 
3 and wait for this comand to return. If this does not 
occur then it is apparent that all 3 rings are broken. 

The most probable cause of an interruption cf all 
three rings is that one node, rather than three independent 
ring segments, has been destroyed. In the example the 
logical conclusion is that node 3 is at fault. A ccmmand 
cculd be sent out by node 5, on ring 1, and addressed to 
node 2, telling it alone to switch to ring 2. This wculd 
bypass the presumed defective node 3. Again the return of 
the message to node 6 would indicate that the ring had teen 
reestablished. Reinitializing the ring in this case wculd be 
more complex, as a node and its processes have been lost. 
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failure cf this last message to returo io a 
reasonable time could initiate further attempts tp 
reconfigure the rirg. Perhaps node 1 could switch to ring 3 
bypassing both nodes 2 and 3. Eventually the processor 
controlling the reconfiguration will exhaust all programmed 
possibilities. It would then send out a message identifying 
the known servicable nodes, in the example only node 4 and 
itself. Any display downstream would receive this message 
and display it. Any processor downstream would add the 
numbers cf the intervening nodes. Thus the last display 
before the break wpuld have as complete a specification as 
possible cf the contiguous segment of the ring. In the 
example ring the display at node 7 would display that nodes 
4 and 5 were serviceable and the display at node 2 would 
display that nodes through 11 and 1 were serviceable. Human 
repair of the ring wpuld now be required. 

It wculd- be possible, during the attempted 
reconfiguration and after reconfiguration had failed, for 
the ccntrclling processor to put control tokens on the ring 
at regular time intervals allowing functioning processes in 
the contiguous portion of the ring to time-share the hus in 
a degraded mode in which a node would not expect to receive 
its own message back. Priority real-time messages could be 
sent as usual and, of course, reconfiguration control 
messages wculd be sent as priorities. Thus one-way 
communication with processes downstream and before the break 
wculd be possible. Hhether this mode would have any real 
value is debatable a-nd would depend on the physical location 
of processes at the time of the break. 

The actual number and routing of inteincdal 
connections wculd depend on the degree cf survivability 
desired, the physical location of the various nodes etc. It 
is obviously very dependent on the particular installation 
and could provide a major area for operational analysis to 
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det€i:iiiiD€ oftinuni values of these parameters. However, only 
the oaxiaum possible cumber of interconnections per node 
would affect the software and once this was decided the 
actual routing would not affect the system. Thus a standard 
software package could handle ring reconfiguration. 


B. BESOIIING CESIGN CONCEPTS 


Appendix I certains a block diagram of a ring interface and 
repeater adapted from those proposed by Harris in fief. 1. 
The main chances are: 

1. the additipn to the repeater cf n continuously 
mcritcred inputs and c selectable outputs, 

2. the additipn of a priority interrupt (pri) and a 

ring break detected token, 

3. the inclus^cn of an m bit delay circuit for 
interrupted message handling, and 

4. the expansion of the input and output buffers to 16 
bits. 


These changes wpuld allow a ring structured distributed 
computing system to operate in the environment of a small 
warship as described in the previous sections. However, 
there are several other areas that should be considered. 
One is the reguirement for mass storage. 

On starting up tie ring or upon restructuring the ring 
after battle damagje, the system must be reestablished and 
available resources distributed as effectively as possible. 
This reguirement pJus the need for any processor to have 
available a large nujnber of processes which it might be 
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reguired tc implement necessitates the existence cf copies 
of these processes available for loading by the processors. 
These programmes m|ght be stored in mass storage devices 
attached directly to the ring and thus be accessible by all 
the processors and provide shared bulk storage as well,. 
However, for survivability there would have to be several 
copies on several separate devices distributed around the 
ring. There is, then, some justification for each processor 
to have its own dedicated bulk storage. This would ensure 
access tc bulk storage no matter what has happened tc the 
ring and would reduce bus traffic during reconfiguration. 
When it is considered that the most probable time that 
leccnfignraticn would be necessary is also the time of 
greatest system activity, i.e. during action, the latter 
consideration could be very important. 

Another area for consideration is the redundancy and 
.autonomy built intp the system. Ideally it would be 
desirable for all processors to have access to all inputs. 
For example, all processors may be capable of performing the 
radar auto-tracking function. However, this would require a 
dedicated bus to each of the radars. It would be 
impractical to provide this to ail processors. Among other 
things, there are not enough input ports tc a processor to 
handle all the inputs reguired. It may be impractical in 
some cases, possibly for reasons of physical location, to 
provide a particular input tc more than one processor. 
Thus, there are gping to be processors dedicated to a 
particular task. Ih^e greater the number of inputs that can 
be made available to more than one processor the greater the 
flexibility there can be for reconfiguration and hence the 
greater the survivability of the system. 

The existence of such dedicated devices in conjunction 
with the modes cf degraded operation have some useful 
consequences. If, fpr example, a display with access to a 
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sensor, a processor, and the appropriate wea 
included in that order in the ring, as long as a 
the ring did not occur between them, the weapon co 
be contrclled and fired. These considerations s 
taken into account when designing bus routings. 


pon were 
break in 
uld still 
hculd be 
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IV. CQNCLUSIC KS AND R5C0MH5NEATI 0N5 


The ccDcept oI using a ring structured distributed 
ccmputing sjstem fpr a general shipboard computing system 
appears tc be feasible. Shile maintaining the advantage of 
flexibility ia size and configuration, the system can be 
extended tc provide the required bus speed, handle real-time 
processes, and cope with the additional hazards of the 
warship environment. The additional software overhead to 
accomplish this does not appear to be too great, however, a 
detailed analysis would be required tc verify this 
assumption. There is some increase in the complexity cf the 
node, but it is still well within the capability cf the 
microprocesscr used by Harris. 

While the system appears feasible there are still many 
areas that require greater study. Neither the software nor 
the ccmaunicaticns overhead of the system have been 
determined. The bus data rates used have been averages. 
Thus a detailed lock at the distribution in frequency and 
length cf interprocessor messages is required and possibly a 
simulaticn needs tc be developed to determine the capability 
of the system to handle peak loads. 

Finally, the ccsts and benefits of the ring structured 
distributed ccmputivng system must be compared with ether 
forms of tie distributed computing system, particularly tne 
bidirecticnal multiply-connected data bus, to determine the 
overall relative value. 
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APPENDIX A 


BINS INTEREACE BLOCK OIAGBANS 
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figure 2 - REPEATER BLOCK CIAGRAH 
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Figure 3 - RING INTERFACE BLOCK DIAGRAM 
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